Papers with k-nearest-neighbor language model
Long-Tail Crisis in Nearest Neighbor Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Prior studies have shown that kNN-LM can retrieve long-tail contexts, leaving the model’s performance underexplored in estimating the probabilities of long-tailed target tokens. |
| Approach: | They investigate the behavior of kNN-LM on low-frequency tokens, examining prediction probability, retrieval accuracy, and token distribution in the datastore. |
| Outcome: | The proposed model improves the perplexity of given text by directly accessing a large datastore built from any text data during inference. |